'\"
'\" Copyright (c) 2015 Ruby Lane
'\"
'\" See the file "LICENSE" for information on usage and redistribution
'\" of this file, and for a DISCLAIMER OF ALL WARRANTIES.
'\"
.TH json n 0.11.0 rl_json "RubyLane/JSON Package Commands"
.so man.macros
.BS
'\" Note:  do not modify the .SH NAME line immediately below!
.SH NAME
json \- Parse, manipulate and produce JSON documents 
.SH SYNOPSIS
.nf
\fBpackage require rl_json\fR ?\fB0.11.0\fR? 

\fBjson get \fIjsonValue\fR ?\fIkey ...\fR?
\fBjson extract \fIjsonValue\fR ?\fIkey ...\fR?
\fBjson exists \fIjsonValue\fR ?\fIkey ...\fR?
\fBjson set \fIjsonVariableName\fR ?\fIkey ...\fR? \fIvalue\fR
\fBjson unset \fIjsonVariableName\fR ?\fIkey ...\fR?
\fBjson foreach \fIvarlist1 jsonValue1\fR ?\fIvarlist2 jsonValue2 ...\fR? \fIscript\fR
\fBjson lmap \fIvarlist1 jsonValue1\fR ?\fIvarlist2 jsonValue2 ...\fR? \fIscript\fR
\fBjson amap \fIvarlist1 jsonValue1\fR ?\fIvarlist2 jsonValue2 ...\fR? \fIscript\fR
\fBjson omap \fIvarlist1 jsonValue1\fR ?\fIvarlist2 jsonValue2 ...\fR? \fIscript\fR
\fBjson string \fIvalue\fR
\fBjson number \fIvalue\fR
\fBjson boolean \fIvalue\fR
\fBjson object \fI?key value ?key value ...??\fR
\fBjson array \fIelem ...\fR
\fBjson bool \fIvalue\fR
\fBjson normalize \fIjsonValue\fR
\fBjson pretty \fIjsonValue\fR
\fBjson template \fIjsonValue\fR ?\fIdictionary\fR?
\fBjson isnull \fIjsonValue\fR ?\fIkey ...\fR?
\fBjson type \fIjsonValue\fR ?\fIkey ...\fR?
\fBjson length \fIjsonValue\fR ?\fIkey ...\fR?
\fBjson keys \fIjsonValue\fR ?\fIkey ...\fR?
\fBjson decode \fIbytes\fR ?\fIencoding\fR?
\fBjson valid ?\fB-extensions\fR \fIextensionlist\fR? ?\fB-details\fR \fIdetailsvar\fR? \fIjsonValue\fR
.fi
.BE
.SH DESCRIPTION
.PP
This package adds a command \fBjson\fR to the interpreter, and defines a new
Tcl_Obj type to store the parsed JSON document.  The \fBjson\fR command
directly manipulates values whose string representation is valid JSON, in a
similar way to how the \fBdict\fR command directly manipulates values whose
string representation is a valid dictionary.  It is similar to \fBdict\fR in
performance.
.TP
\fBjson get \fIjsonValue\fR ?\fIkey ...\fR?
.
Extract the value of a portion of the \fIjsonValue\fR, returns the closest
native Tcl type (other than JSON) for the extracted portion. The \fIkey ...\fR
arguments are a path, as described in \fBPATHS\fR below.
'\" TODO: describe what happens with a null
.TP
\fBjson extract \fIjsonValue\fR ?\fIkey ...\fR?
.
Extract the value of a portion of the \fIjsonValue\fR, returns the JSON
fragment. The \fIkey ...\fR arguments are a path, as described in \fBPATHS\fR
below.
.TP
\fBjson exists \fIjsonValue\fR ?\fIkey ...\fR?
.
Tests whether the supplied key path (see \fBPATHS\fR below) resolves to
something that exists in \fIjsonValue\fR (i.e., that it can be used with
\fBjson get\fR without error) and is not null.  Returns false if the value
named by the path \fIkey ...\fR is null.
.TP
\fBjson set \fIjsonVariableName\fR ?\fIkey ...\fR? \fIvalue\fR
.
Updates the JSON value stored in the variable \fIjsonVariableName\fR,
replacing the value referenced by \fIkey ...\fR (a path as described in
\fBPATHS\fR below) with the JSON value \fIvalue\fR.  If \fIvalue\fR is
a valid JSON as given by the JSON grammar, it is added as that JSON type,
otherwise it is converted to a JSON string.  Thus the following are equivalent
(modulo efficiency):

.CS
 json set doc foo [json string baz]
 json set doc bar [json number 123]
 json set doc baz [json boolean true]

 #------------------------------------------
 json set doc foo baz
 json set doc bar 123
 json set doc baz true
.CE

Watch out for unintended behaviour if the value might look like a boolean or
number but not meet the JSON grammar for those types, in which case the value
is converted to a JSON string:

.CS
 json set doc foo [json boolean yes]
 # Key "foo" contains the JSON boolean value "true"

 json set doc foo yes
 # Key "foo" contains the JSON string value "yes"
.CE

Constructing the values using [\fBjson \fItype\fR] forces the conversion to
the specified JSON type, or throws an exception if that can't be done.
Which is more efficent will depend on the situation:

.CS
 set doc {[]}
 for {set i 0} {$i < 100} {incr i} {
 	json set doc end+1 [json boolean true]	;# 1
 	json set doc end+1 true				;# 2
 }
 # 2 will be faster since "true" will be stored as a literal, and converted
 # to a JSON boolean.  Each loop iteration will just append another reference
 # to this static value to the array, whereas 1 will call [json boolean] each
 # iteration.
 
 set doc {[]}
 for {set i 0} {$i < 100} {incr i} {
 	json set doc end+1 [json string false$i]	;# 1
 	json set doc end+1 false$i				;# 2
 }
 # 1 will be faster since [json string] knows what the type is and directly
 # creates the new element as that type.  2 Needs to parse the string to
 # determine the type.
.CE
.TP
\fBjson unset \fIjsonVariableName\fR ?\fIkey ...\fR?
.
Updates the JSON value stored in the variable \fIjsonVariableName\fR, removing
the value referenced by \fIkey ...\fR, a path as described in \fBPATHS\fR
below.  If the path names a entry in an object then that key is removed from the
object.  If the path names an element in an array, that element is removed
and all later elements are moved up.
.TP
\fBjson template \fIjsonValue\fR ?\fIdictionary\fR?
.
Return a JSON value by interpolating the values from \fIdictionary\fR into the
template, or from variables in the current scope if \fIdictionary\fR is not
supplied, in the manner described in the section \fBTEMPLATES\fR.
.TP
\fBjson string \fIvalue\fR
.
Return a JSON string with the value \fIvalue\fR.
.TP
\fBjson number \fIvalue\fR
.
Return a JSON number with the value \fIvalue\fR.
.TP
\fBjson boolean \fIvalue\fR
.
Return a JSON boolean with the value \fIvalue\fR.  Any of the forms accepted by
Tcl_GetBooleanFromObj are accepted and normalized.
.TP
\fBjson object \fI?key value ?key value ...??\fR -or- \fBjson object \fIpacked_value\fR
.
Return a JSON object with the each of the keys and values given.  \fIvalue\fR is a list
of two elements, the first being the type {string, number, boolean, null, object, array, json},
and the second being the value.  The alternate syntax \fBjson object \fIpacked_value\fR
takes the list of keys and values as a single arg instead of a list of args, but is
otherwise the same.
.TP
\fBjson array \fI?elem ...?\fR
.
Return a JSON array containing each of the elements given.  \fIelem\fR is a list
of two elements, the first being the type {string, number, boolean, null, object, array, json},
and the second being the value.
.TP
\fBjson foreach \fIvarList1 jsonValue1\fR ?\fIvarList2 jsonValue2 ...\fR? \fIscript\fR
.
Evaluate \fIscript\fR in a loop in a similar way to the \fBforeach\fR command.
In each iteration, the values stored in the iterator variables in each
\fIvarList\fR are the JSON fragments from \fIjsonValue\fR. This command
supports iterating over JSON arrays and JSON objects.  In the JSON object
case, the corresponding \fIvarList\fR must be a two element list, with the
first specifiying the variable to hold the key and the second the value.  In
the JSON array case, the rules are the same as the \fBforeach\fR command.
.TP
\fBjson lmap \fIvarList1 jsonValue1\fR ?\fIvarList2 jsonValue2 ...\fR? \fIscript\fR
.
As for \fBjson foreach\fR, except that it is collecting; the result from each
evaluation of \fIscript\fR is added to a Tcl list and returned as the result
of the \fBjson lmap\fR command.  If the \fIscript\fR results in a TCL_CONTINUE
code (e.g., the script does \fBcontinue\fR), that iteration is skipped and no
element is added to the result list.  If it results in TCL_BREAK (e.g., the
script does \fBbreak\fR) the iterations are stopped and the results
accumulated so far are returned.
.TP
\fBjson amap \fIvarList1 jsonValue1\fR ?\fIvarList2 jsonValue2 ...\fR? \fIscript\fR
.
As for \fBjson lmap\fR, but the result is a JSON array rather than a list.  If
the result of each iteration is a JSON value it is added to the array as-is,
otherwise it is converted to a JSON string.
.TP
\fBjson omap \fIvarList1 jsonValue1\fR ?\fIvarList2 jsonValue2 ...\fR? \fIscript\fR
.
As for \fBjson lmap\fR, but the result is a JSON object rather than a list.
The result of each iteration must be a dictionary (or a list of 2n elements,
including n = 0).  Tcl_ObjType snooping is done to ensure that the iteration
over the result is efficient for both dict and list cases.
Each entry in the dictionary will be added to the result object.  If the value
for each key in the iteration result is a JSON value it is added to the array
as-is, otherwise it is converted to a JSON string.
.TP
\fBjson isnull \fIjsonVariableName\fR ?\fIkey ...\fR?
.
Return a boolean indicating whether the named JSON value is null.
.TP
\fBjson type \fIjsonVariableName\fR ?\fIkey ...\fR?
.
Return the type of the named JSON value, one of "object", "array", "string",
"number", "boolean" or "null".
.TP
\fBjson length \fIjsonVariableName\fR ?\fIkey ...\fR?
.
Return the length of the of the named JSON array, number of entries in the
named JSON object, or number of characters in the named JSON string.  Other
value types aren't supported.
.TP
\fBjson keys \fIjsonVariableName\fR ?\fIkey ...\fR?
.
Return the keys in the of the named JSON object, found by following the path of \fIkey\fRs.
.TP
\fBjson normalize \fIjsonValue\fR
.
Return a
.QW normalized
version of the input \fIjsonValue\fR, i.e., with all optional whitespace
trimmed.
.TP
\fBjson pretty \fIjsonValue\fR
.
Returns a pretty-printed string representation of \fIjsonValue\fR.  Useful for
debugging or inspecting the structure of JSON data.
.TP
\fBjson decode \fIbytes\fR ?\fIencoding\fR?
.
Rl_json operates on characters, as returned from Tcl's Tcl_GetStringFromObj,
not raw bytes, so considerations of encoding are strictly outside of its scope
(other than ignoring a byte order mark if the string starts with one).  The
JSON RFC lays out some behaviour for conforming implementations regarding
character encoding, and ensuring that an application using rl_json meets that
standard would be up to the application.  Some aspects are not straightforward,
so rl_json provides this utility subcommand that takes binary data in \fIbytes\fR
and returns a character string according to the RFC specified behaviour.  If
the optional \fIencoding\fR argument is given, that encoding will be used to
interpret \fIbytes\fR.  The supported encodings are those specified in the RFC:
utf-8, utf-16le, utf-16be, utf-32le, utf-32be.  If the string starts with a BOM
(byte order mark (U+FFFE)), and no encoding is given, it will be determined
from the encoding of the BOM.  All the encodings listed are supported, even if
Tcl lacks support for the utf-16 and utf-32 encodings natively.  However,
without native support the conversion will be slow.

This might look something like this in an application:

.CS
 proc readjson file {
 	set h [open $file rb]	;# Note that the file is opened in binary mode - no encoding
 	try {
 		json decode [read $h]
 	} finally {
 		close $h
 	}
 }
.CE
.TP
\fBjson valid\fR ?\fB-extensions\fR \fIextensionlist\fR? ?\fB-details\fR \fIdetails\fR? \fIjsonValue\fR
.
Validate \fBjsonValue\fR against the JSON grammar, returning true if it
conforms and false otherwise.  A list of extensions to accept can be supplied
with \fB-extensions\fR, with only one currently supported extension:
\fBcomments\fR, which accepts JSON documents containing \fB// foo\fR and \fB/*
foo */\fR style comments anywhere whitespace would be valid.  To reject
documents containing comments, set \fIextensionlist\fR to {}.

Validation using this subcommand is about 3 times faster than parsing and
catching a parsing exception, and it allows strict validation against the RFC
without comments.

If validation fails and \fB-details\fR \fIdetailsvar\fR is supplied, the variable
\fIdetailsvar\fR is set to a dictionary containing the keys:
.RS 10
.IP \fBerrmsg\fR 10
A reason for the failure.
.IP \fBdoc\fR 10
The document that failed validation
.IP \fBchar_ofs\fR 10
The character offset into \fBdoc\fR that caused validation to fail.
.RE
.SH PATHS
.PP
Several of the commands (e.g., \fBjson get\fR, \fBjson exists\fR, \fBjson
set\fR and \fBjson unset\fR) accept a path specification that names some
subset of the supplied \fIjsonValue\fR.  The rules are similar to the
equivalent concept in the \fBdict\fR command, except that the paths used by
\fBjson\fR allow indexing into JSON arrays by the integer key (or a string
matching the regex
.QW "^end(-[0-9]+)?$" ).
.SH TEMPLATES
.PP
The command \fBjson template\fR generates JSON documents by interpolating
values into a template from a supplied dictionary or variables in the current
call frame, a flexible mechanism for generating complex documents.  The
templates are valid JSON documents containing string values which match the
regex
.QW "^~[SNBJTL]:.+$" .
The second
character determines what the resulting type of the substituted value will be:
.RS 3
.IP \fBS\fR 3
A string.
.IP \fBN\fR 3
A number.
.IP \fBB\fR 3
A boolean.
.IP \fBJ\fR 3
A JSON fragment.
.IP \fBT\fR 3
A JSON template (substitutions are performed on the inserted fragment).
.IP \fBL\fR 3
A literal. The resulting string is simply everything from the fourth character
onwards (this allows literal strings to be included in the template that would
otherwise be interpreted as the substitutions above).
.RE
.PP
None of the first three characters for a template may be escaped.
.PP
The value inserted is determined by the characters following the substitution
type prefix.  When interpolating values from a dictionary they name keys in the
dictionary which hold the values to interpolate.  When interpolating from
variables in the current scope, they name scalar or array variables which hold
the values to interpolate.  In either case if the named key or variable doesn't
exist, a JSON null is interpolated in its place.
.SH EXCEPTIONS
.PP
Exceptions are thrown when attempting to parse a string which isn't valid JSON,
or when a named path is invalid or doesn't exist:
.TP
\fBRL JSON PARSE \fIerrormessage\fR \fIstring\fR \fIcharOfs\fR
.
Thrown when trying to parse a string that isn't valid JSON.  The \fIstring\fR
element contains the string that failed to parse, and the first invalid character
is at offset \fIcharOfs\fR within that string, using 0 based offsets.
.TP
\fBRL JSON BAD_PATH \fIpath\fR
.
Thrown when indexing into a JSON value and the specified path isn't valid.
\fIpath\fR is the left subset of the path up to first element that caused the
failure.
.SH EXAMPLES
.PP
Produce a JSON value from a template:
.PP
.CS
 \fBjson template\fR {
     {
         "thing1": "~S:val1",
         "thing2": ["a", "~N:val2", "~S:val2", "~B:val2",
                    "~S:val3", "~L:~S:val1"],
         "subdoc1": "~J:subdoc",
         "subdoc2": "~T:subdoc"
     }
 } {
     val1   hello
     val2   1e6
     subdoc {
         { "thing3": "~S:val1" }
     }
 }
.CE
.PP
The result (with formatting for readability):
.PP
.CS
 {
     "thing1":"hello",
     "thing2":["a",1000000.0,"1e6",true,null,"~S:val1"],
     "subdoc1":{"thing3":"~S:val1"},
     "subdoc2":{"thing3":"hello"}
 }
.CE
.PP
Incrementally append an element to an array (similar to \fBdict lappend\fR):
.PP
.CS
 set doc {{"foo":[]}}
 for {set i 0} {$i < 4} {incr i} {
     json set doc foo end+1 [json string "elem: $i"]
 }
 # $doc is {"foo":["elem 0","elem 1","elem 2","elem 3"]}
.CE
.PP
Similar to the above, but prepend the elements instead:
.PP
.CS
 set doc {{"foo":[]}}
 for {set i 0} {$i < 4} {incr i} {
     json set doc foo -1 [json string "elem: $i"]
 }
 # $doc is {"foo":["elem 3","elem 2","elem 1","elem 0"]}
.CE
.PP
Trim an element out of an array:
.PP
.CS
 set doc {["a","b","c"]}
 json unset doc 1
 # $doc is {["a","c"]}
.CE
.PP
Implicitly create objects when setting a path that doesn't exist:
.PP
.CS
 set doc {{"foo":1}}
 json set doc bar baz {"hello, new obj"}
 # $doc is {"foo":1,"bar":{"baz":"hello, new obj"}}
.CE
.PP
Index through objects and arrays (the path elements are unambiguous because the
json types they index into are known at resolve time):
.PP
.CS
 set doc {{"foo":["a",{"primes":[2,3,5,7,11,13,17,19]},"c"]}}
 json get $doc foo 1 primes end-1
 # returns 17
.CE
.PP
Handle a parse error and display a helpful message indicating the character
that caused the failure:
.PP
.CS
 try {
     json get {
         {
             "foo": {
                 "bar": true,
             }
         }
     } foo bar
 } trap {RL JSON PARSE} {errmsg options} {
     lassign [lrange [dict get $options -errorcode] 4 5] doc char_ofs
     puts stderr "$errmsg\\n[string range $doc 0 $char_ofs-1](here -->)[string range $doc $char_ofs end]"
 }
.CE
.PP
Produces:
.PP
.CS
 Error parsing JSON value: Illegal character at offset 37
 
 		{
 			"foo": {
 				"bar": true,
 			(here -->)}
 		}
.CE
'\" .SH "SEE ALSO"
'\" dict(n), list(n)
.SH KEYWORDS
json, parsing, formatting
'\" Local Variables:
'\" mode: nroff
'\" End:
